Content Aware Neural Style Transfer

In the emerging inter-disciplinary field of art and image processing, algorithms have been developed to assist the analysis of art work. In most applications, especially brush stroke analysis, high resolution digital images of paintings are required to capture subtle patterns and details in the high frequency range of the spectrum. Algorithms have been developed to learn styles of painters from their digitized paintings to help identify authenticity of controversial paintings. However, high quality testing datasets containing both original and forgery are limited to confidential image files provided by museums, which is not publicly available, and a small sets of original/copy paintings painted by the same artist, where copies were deferred to two weeks after the originals were finished. Up to date, no synthesized painting by computers from a real painting has been used as a negative test case, mainly due to the limitation of prevailing style transfer algorithms. There are two main types of style transfer algorithms, either transferring the tone (color, contrast, saturation, etc.) of an image, preserving its patterns and details, or distorting the texture uniformly of an image to create “style”. In this paper, we are interested in a higher level of style transfer, particularly, transferring a source natural image (e.g. a photo) to a high resolution painting given a reference painting of similar object. The transferred natural image would have a similar presentation of the original object to that of the reference painting. In general, an object is painted in a different style of brush strokes than that of the background, hence the desired style transferring algorithm should be able to recognize the object in the source natural image and transfer brush stroke styles in the reference painting in a content-aware way such that the styles of the foreground and the background, and moreover different parts of the foreground in the transferred image, are consistent to that in the reference painting. Recently, an algorithm based on deep convolutional neural network has been developed to transfer artistic style from an art painting to a photo [2]. Successful as it is in transferring styles from impressionist paintings of artists such as Vincent van Gogh to photos of various scenes, the algorithm is prone to distorting the structure of the content in the source image and introducing artifacts/new